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ABSTRACT 



Techniques for protecting the security of digital 
representations, and of analog forms made from them are 
presented. The techniques include authentication techniques 
that can authenticate both a digital representation and an 
analog form produced from the digital representation, an 
active watermark that contains program code that may be 
executed when the watermark is read, and a watermark agent 
that reads watermarks and sends messages with information 
concerning the digital representations that contain the water- 
marks. The authentication techniques use semantic informa- 
tion to produce authentication information. Both the seman- 
tic information and the authentication information survive 
when an analog form is produced from the digital represen- 
tation. In one embodiment, the semantic information is 
alphanumeric characters and the authentication information 
is either contained in a watermark embedded in the digital 
representation or expressed as a bar code. With the active 
watermark, the watermark includes program code. When a 
watermark reader reads the watermark, it may cause the 
program code to be executed. One application of active 
watermarks is making documents that send messages when 
they are operated on. A watermark agent may be either a 
permanent resident of a node in a network or of a device 
such as a copier or it may move from one network node to 
another. In the device or node, the watermark agent executes 
code which examines digital representations residing in the 
node or device for watermarked digital representations that 
are of interest to the watermark agent. The watermark agent 
then sends messages which report the results of its exami- 
nation of the digital representations. If the watermarks are 
active, the agent and the active watermark may cooperate 
and the agent may cause some or all of the code than an 
active watermark contains to be executed. 

56 Claims, 12 Drawing Sheets 
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// Instantiate a file filter. 

FileFilter filePilter =new PilePilter();> 

// filter out all image files from the file system. 

String [] filenames = filePilter. filterlmagesO 

// get host environment information. 1207 

Envlnfo env = getEnvInfoO 1209 

// construct a new vector used to store action results. 

Vector results = new Vector ();-. 

v 1211 

forUnt i=0; i< filenames. length; i++) { 
// check each image file for watermark. 
String watermark = checkWatermark (filenames [i] j-/-" 1215 
if (watermark != null) { // if a watermark is found ^ 

// match the watermark with the host environment. 
String match = matchEnv (watermark, env) ;-/-1219 
// take action according to the matching result, 



>1203 



J 



> 
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String result = takeAction (match) ; 
// add the result to the vector, 
results. addElement (result) ; 



^1221 
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ACTIVE WATERMARKS AND WATERMARK 
AGENTS 

CROSS REFERENCE TO RELATED PATENT 5 
APPLICATIONS 

This application has the same Detailed Description as Jian 
Zhao and E. Koch, Digital Authentication with Analog 
Documents, assigned to Mediasec LLP and filed on even 
date with this application. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The invention relates generally to digital representations 15 
of images and other information and more specifically to 
techniques for protecting the security of digital representa- 
tions and of analog forms produced from them. 

2. Description of the Prior Art 

Nowadays, the easiest way to work with pictures or 20 
sounds is often to make digital representations of them. 
Once the digital representation is made, anyone with a 
computer can copy the digital representation without 
degradation, can manipulate it, and can send it virtually 
instantaneously to anywhere in the world. The Internet, 25 
finally, has made it possible for anyone to distribute any 
digital representation from anywhere in the world From the 
point of view of the owners of the digital representations, 
there is one problem with all of this: pirates, too, have 
computers, and they can use them to copy, manipulate, and 30 
distribute digital representations as easily as the legitimate 
owners and users can. If the owners of the original digital 
representations are to be properly compensated for making 
or publishing them, the digital representations must be 
protected from pirates. There are a number of different 35 
approaches that can be used: 

the digital representation may be rendered unreadable 
except by its intended recipients; this is done with 
encryption techniques; 
the digital representation may be marked to indicate its 

authenticity; this is done with digital signatures; 
the digital representation may contain information from 
which it may be determined whether it has been tam- 
pered with in transit; this information is termed a digest 45 
and the digital signature often includes a digest; 
the digital representation may contain a watermark, an 
invisible indication of ownership which cannot be 
removed from the digital representation and may even 
be detected in an analog copy made from the digital 50 
representation; and 
the above techniques can be employed in systems that not 
only protect the digital representations, but also meter 
their use and/or detect illegal use. 
For an example of a system that uses encryption to protect 55 
digital representations, see U.S. Pat. No. 5,646,999, Saito, 
Data Copyright Management Method, issued Jul. 8, 1997, 
for a general discussion of digital watermarking, see Jian 
Zhao, "Look, It's Not There", in: BYTE Magazine, January, 
1997, Detailed discussions of particular techniques for digi- 60 
tal watermarking may be found in E. Koch and J. Zhao, 
"Towards Robust and Hidden Image Copyright Labeling", 
in: Proc. Of 1995 IEEE Workshop on Nonlinear Signal and 
Image Processing, Jun. 20-22, 1995, and in U.S. Pat. No. 
5,710,834, Rhoads, Method and Apparatus Responsive to a 65 
Code Signal Conveyed through a Graphic Image, issued Jan. 
20, 1998. For an example of a commercial watermarking 
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system that uses the digital watermarking techniques dis- 
closed in the Rhoads patent, see Digimarc Watermarking 
Guide, Digimarc Corporation, 1997, available at in March, 
1998 at http://www.digimarc.com. 

FIG. 1 shows a prior-art system 101 which employs the 
above protection techniques. A number of digital represen- 
tation clients 105, of which only one, digital representation 
client 105(/) is shown, are connected via a network 103 such 
as the Internet to a digital representation server 129 which 
receives digital representations from clients 105 and distrib- 
utes them to clients 105. Server 129 includes a data storage 
device 133 which contains copied digital representations 
135 for distribution and a management data base 139. Server 
129 further includes a program for managing the digital 
representations 135, a program for reading and writing 
watermarks 109, a program for authenticating a digital 
representation and confirming that a digital representation is 
authentic 111, and a program for encrypting and decrypting 
digital representations 113. Programs 109, 111, and 113 
together make up security programs 107. 

Client 105 has its own versions of security programs 107, 
it further has editor/viewer program 115 which lets the user 
of client 105 edit and/or view digital representations that it 
receives via network 103 or that are stored in storage device 
117. Storage device 117 as shown contains an original digital 
representation 119 which was made by a user of client 105 
and a copied digital representation 121 that was received 
from DR Server 129. Of course, the user may have made 
original representation 119 by modifying a copied digital 
representation. Editor/viewer program 115, finally, permits 
the user to output digital representations to analog output 
devices 123. Included among these devices are a display 
123, upon which an analog image 124 made from a digital 
representation may be displayed and a printer 127 upon 
which an analog image 126 made from the digital represen- 
tation may be printed, A loudspeaker may also be included 
in analog output devices 123. The output of the analog 
output device will be termed herein an analog form of the 
digital representation. For example, if the output device is a 
printer, the analog form is printed sheet 126; if it is a display 
device, it is display 124. 

When client 105(/) wishes to receive a digital represen- 
tation from server 129, it sends a message requesting the the 
digital representation to server 129. The message includes at 
least an identification of the desired digital representation 
and an identification of the user. Manager 131 responds to 
the request by locating the digital representation in CDRs 
135, consulting management data base 139 to determine the 
conditions under which the digital representation may be 
distributed and the status of the user of client 105 as a 
customer. If the information in data base 139 indicates to 
manager 131 that the transaction should go forward, man- 
ager 131 sends client 105(/) a copy of the selected digital 
representation. In the course of sending the copy, manager 
131 may use watermark reader/writer 109 to add a water- 
mark to the digital representation, use authenticator/ 
confirmer 111 to add authentication information, and 
encrypter/decrypter 113 to encrypt the digital representation 
in such a fashion that it can only be decrypted in DR client 
1050). 

When client 105(f) receives the digital representation, it 
decrypts it using program 113, confirms that the digital 
representation is authentic using program 111, and editor/ 
viewer 115 may use program 109 to display the watermark. 
The user of client 105(/) may save the encrypted or unen- 
crypted digital representation in storage 117. The user of 
client 105(/) may finally employ editor/viewer 115 to decode 
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the digital representation and output the results of the representation may send a message to a billing server, 

decoding to an analog output device 123. Analog output Another use is destroying the digital representation if a user 

device 123 may be a display device 125, a printer 127, or in attempts an operation for which the user has no privileges, 

the case of digital representations of audio, a loudspeaker. In this aspect, the invention includes apparatus and methods 

It should be pointed out that when the digital represen- 5 for making and reading active watermarks. The methods and 

tation is displayed or printed in analog form, the only apparatus for making active watermarks may be employed 

remaining protection against copying is watermark 128, anywhere present-day watermark makers are employed, and 

which cannot be perceived in the analog form by the human the methods and apparatus for reading active watermarks 

observer, but which can be detected by scanning the analog may be employed anywhere present-day watermark readers 

form and using a computer to find watermark 128. Water- 10 are employed. 

mark 128 thus provides a backup to encryption: if a digital i n another aspect, the invention is a watermark agent 
representation is pirated, either because someone has broken which is located in a device upon which digital representa- 
the encryption, or more likely because someone with legiti- tions containing watermarks are resident. The watermark 
mate access to the digital representation has made illegiti- a g en t reads the watermark in a digital representation and 
mate copies, the watermark at least makes it possible to is performs actions ranging from sending a message to the user 
determine the owner of the original digital representation through sending a message to a monitoring agent, moving 
and given that evidence, to pursue the pirate for copyright the digital representation, or changing its access rights to 
infringement and/or violation of a confidentiality agreement. destroying the digital representation. Some watermark 
If the user of client 105(/) wishes to send an original agents are mobile. A mobile watermark agent moves from 
digital representation 119 to DR server 129 for distribution, 20 node to node in a network. In each node, it examines the 
editor/viewer 115 will send digital representation 119 to watermarks on digital representations stored in the node and 
server 129. In so doing, editor/viewer 115 may use security se nds messages reporting its findings to a monitoring agent 
programs 107 to watermark the digital representation, located in the network. When a watermark agent encounters 
authenticate it, and encrypt it so that it can be decrypted only a digital representation with an active watermark, the water- 
by DR Server 129, Manager 131 in DR server 129 will, 25 ma rk agent may execute the program code contained in the 
when it receives digital representation 119, use security active watermark. 

programs 107 to decrypt digital representation 119, confirm 0ther objects and advantages of the invention will be 

its authenticity, enter information about it in management apparent to those skilled in the arts to which the invention 

data base 139, and store it in storage 133. pertains upon perusing the following Detailed Description 

In the case of the Digimarc system referred to above, 30 ^jjj Drawing, wherein: 
manager 131 also includes a World Wide Web spider, that is, 

a program that systematically follows World Wide Web links BRIEF DESCRIPTION OF THE DRAWING 

such as HTTP and FTP links and fetches the material pointed FIG. 1 is a block diagram of a prior-art system for 

to by the links. securely distributing digital representations; 

Manager program 131 uses watermark reading/writing 35 - . ,. f n , , 4 r . 

, r T . , j •£ . i ♦ i • FIG. 2 is a diagram of a first embodiment of an analog 

program to read any watermark, and if the watermark is c , . j 

f t .j^L^m in form that can be authenticated; 

known to management database 139, manager program 131 ' 

takes whatever action may be required, for example, deter- FIG - 3 1S a diagram of a second embodiment of an analog 

mining whether the site from which the digital representa- form that caD be authenticated; 

tion was obtained has the right to have it, and if not, 40 FIG- 4 is a diagram of a system for adding authentication 

notifying the owner of the digital representation. information to an analog form; 

While encryption, authentication, and watermarking have FIG. 5 is a diagram of a system for authenticating an 

made it much easier for owners of digital representations to analog form; 

protect their property, problems still remain. One such FIG. 6 is a diagram of a system for making an active 

problem is that the techniques presently used to authenticate 45 watermark; 

digital documents do not work with analog forms; FIG. 7 is an example of code from an active watermark; 

consequently, when the digital representation is output in r> ir ^ 0 . c c tU , . 

. n J ' , , ? . K . A . , FIG. 8 is a diagram of a system for executing the code in 

analog form, the authentication is lost. Another is that . , 

« ^ . j. i • an active watermark; 

present-day systems for managing digital representations are „^ A . _ _ , . 

not flexible enough. A third is that watermark checking such 50 9 15 a dli & im of a s y st6m for Pacing a watermark 

as that done by the watermark spider described above is a 8 ent > 

limited to digital representations available on the Internet. It FIG - 10 is a diagram of a system for receiving a water- 
is an object of the present invention to overcome the above mark agent; 

problems and thereby to provide improved techniques for FIG. 11 is a detailed diagram of access information 603; 

distributing digital representations. 55 and 

SUMMARY OF THE INVENTION F, f G - 12 15 m CXample ° f Code CXCCU,ed by 8 wateraJark 

agent. 

In one aspect, the invention is an active watermark, that The reference numbers in the drawings have at least three 
is, a watermark in which the information included in the digits. The two rightmost digits are reference numbers 
watermark includes program code that can be executed 60 within a figure, the digits to the left of those digits are the 
when the watermark is read. What the program code does is number of the figure in which the item identified by the 
of course completely arbitrary. For example, the code in the reference number first appears. For example, an item with 
active watermark can send a message each time a particular reference number 203 first appears in FIG. 2. 
operation is performed on the digital representation contain- 
ing the active watermark. One use for such an active 65 DETAILED DESCRIPTION 
watermark is for billing: each time a digital representation The following Detailed Description will first disclose a 
with an active watermark is copied, for instance, the digital technique for authenticating digital representations that sur- 
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vives output of an analog form of the digital representation, Watermarking changes the digital representation; 

will then disclose active watermarks, that is, watermarks that consequently, if a digital representation is watermarked 

contain programs, and will finally disclose watermark after me original digest is made, the watermarking 

agents, that is, programs which examine the digital water- invalidates the original digest, i.e., it is no longer 

marks on digital representations stored in a system and 5 comparable with the new digest that the recipient 

. i i i * j * i * ,i_ . u * j makes from the watermarked document, 

thereby locate digital representations that are being used „ t , •« •_ . * • ■ 

i More troublesome still, when a digital representation is 

improper y. output in analog form, so much information about the 

Authentication that is Preserved in Analog Forms: FIGS. digital representation is lost that the digital represen- 

2-5 10 tation cannot be reconstructed from the analog form. 

Digital representations are authenticated to make sure that ™ us > ^^^"Jf^^^c ^w df eV^ ^the 

they have not been altered in transit. Alteration can occur as wa ^ 0 P TO UC1 ^ ^ ara ] ^ es rom 

. . . , , . , analog torm. 

a result of transmission errors that occur during the course what fc needed tQ overcome these prob i ems is an authen . 

of transmission from the source of the digital representation tication technique which uses information for authentication 

to its destination, as a result of errors that arise due to is w bich is independent of the particular form of the digital 

damage to the storage device being used to transport the representation and which will be included in the analog form 

digital representation, as a result of errors that arise in the when the analog form is output. As will be explained in more 

course of writing the digital representation to the storage detail in the following, the first requirement is met by 

device or reading the digital representation from the storage selecting semantic information from the digital representa- 

device, or as a result of human intervention. A standard 20 tion and usin 8 on j v the semantic information to make the 

technique for authentication is to make a digest of the digital digest. The second requirement is met by incorporating the 

representation and send the digest to the destination together dl 8f "»° *» d 'S' tal representation ,n a fashion such that ,t 

with the digital representation. At the destination, another 00 < he one , b\nd does not affect the semantic information 

digest is made from the digital representation as received used to make the and ° n he 0,h " hand su ™ ve , s m , ,he 

and compared with the first. If they are the same, the digital 25 ana '°g form u . In u ,he cas f of d «™*> an aulhent.cat.on 

representation has not changed. The digest is simply a value ««* n "l M meets these requirements can be used no 

which is much shorter than the digital representation but is °^ l0 , auth « nticate f analog forms of documents that exist 

related to it such that any change in the digital representation Pf"°anly in digital form, but also to authenticate documents 

will with very high probability result in a change to the that ex f f™ 3 " 1 * °« ' ^ m analo S form > for exam P le 

... ' e r ' ° 30 paper checks and identification cards. 

8 . ... ... Semantic Information 

Where human intervention is a serious concern, the digest The xmanlic in f ormal i 0 n in a digital representation is 

is made using a one-way hash function, that is, a Junction , hat rtion of ^ information m th e digital representation 

that produces a digest from which it is extremely difficult or lhat must be m m the analog form made from , he digital 

impossible to learn anything about the input that produced it. 3J re p resen taUon if the human who perceive the analog form is 

The digest may additionally be encrypted so that only the tQ considef it a of ^ QligM {mm whjcn tQe digital 

recipient of the digital representation can read it. A common representation was made. For example, the semantic infor- 

tcchmque is to use the encrypted digest as the digital ma ,i on in a digital representation of an image of a document 

signature for the digital representation, that is, not only to fe the representalions o£ the alphanumeric characters in the 

show that the digital representation has not been altered in 4Q docum ent, where alphanumeric is understood to include 

transit, but also to show that it is from whom it purports to representations of any kind of written characters or punc- 

be from. If the sender and the recipient have exchanged |uatioD marks> inc i u ding those belonging to non-Latin 

public keys, the sender can make the digital signature by alphabetS) t0 syllabic writing systems> and t0 ideographic 

encryptmg the digest with the sender s private key. The wfitm tems Given , he 

alphanumeric characters, the 

recipient can use the sender s public key to decrypt the 45 human red fcnt of the ^ form can determine whether 

digest, and having done that^ the recipient compares the a documem is a of me original) everi the 

digest with the digest made from the received digital rep- characters may have different fonts and may have been 

resentation. If they are not the same, either the digital formatted differe ntly in the original document. There is 

representation has been altered or the digital representation analogous semantic information in digital representations of 

is not from the person to whom the public key used to 5Q ictufes ^ of audio iaformatioil( In the case of pictures> it 

decrypt the digest belongs. For details or .authentication see fc ^ mformation that is required for the human ^ 

Section 3.2 of Bruce Schneier, Applied Cryptography, John perceives the analog form t0 agree mat the analog form is a 

Wiley and Sons, 1994. copy ^ aJbeit a bad Qn ^ of the original picture, and the same 

The only problem with authentication is that it is based ^ the case with audio information, 

entirely on the digital representation The information used to 55 \ a the case of a document written in English, the semantic 

make the digest is lost when the digital representation is information in the document is the letters and punctuation of 

output in analog form. For example, if the digital represen- the document. If the document is in digital form, it may be 

tation is a document, there is no way of determining from a represented either as a digital image or in a text represen- 

paper copy made from the digital representation whether the tat i on language such as those used for word processing or 

digital representation from which the paper copy was made 60 printing. In the first case, optical character recognition 

is authentic or whether the paper copy is itself a true copy (OCR) technology may be applied to the image to obtain the 

of the digital representation. letters and punctuation; in the second case, the digital 

While digital watermarks survive and remain detectable representation may be parsed for the codes that are used to 

when a digital representation is output in analog form, the represent the letters and punctuation in the text representa- 

authentication problem cannot be solved simply by embed- 65 tion language. If the document is in analog form, it may be 

ding the digest or digital signature in the watermark. There scanned to produce a digital image and the OCR technology 

are two reasons for this: applied to the digital image produced by scanning. 



08/19/2004, EAST Version: 1.4.1 



US 6,754,822 Bl 

7 8 

Using Semantic Information to Authenticate an Analog character reading (OCR) device, in the second, it will simply 

Form: FIGS. 2 and 3 parse the document representation looking for character 

Because the semantic information mist he present in the codes, 

analog form, it may be read from the analog form and used In any case, at the end of the process, semantics reader 

to compute a new digest. If the old digest was similarly made 5 405 will have extracted some form of semantic information, 

from the semantic information in the digital representation for example the ASCII codes corresponding to the alphanu- 

and the old digest is readable from the analog form, the new meric characters, from representation 403. This digital infor- 

digest and the old digest can be compared as described in the mation is then provided to digest maker 409, which uses it 

discussion of authentication above to determine the authen- to make semantic digest 411 in any of many known ways, 

ticity of the analog form. 10 Depending on the kind of document the semantic digest is 

FIG. 2 shows one technique 201 for incorporating the old made from and its intended use, the semantic digest may 

digest into an analog form 203. Analog form 203 of course have a form which requires an exact match with the new 

includes semantic information 205; here, analog form 203 is digest or may have a form which permits a "fuzzy" match, 

a printed or faxed document and semantic information 205 Digital representation 403 and semantic digest 411 are then 

is part or all of the alphanumeric characters on analog form 15 provided to digest incorporator 413, which incorporates a 

203. Sometime before analog form 203 was produced, representation 207 of digest 411 into the digital representa- 

semantic information 205 in the digital representation from tion used to produce analog form 203. As indicated above, 

which analog form 203 was produced was used to make the representation must be incorporated in such a way that 

semantic digest 207, which was incorporated into analog it does not affect semantic information 205. Incorporator 413 

form 203 at a location which did not contain semantic 20 then outputs the representation it produces to analog form 

information 205 when analog form 203 was printed. In some producer 415, which produces analog form 203 in the usual 

embodiments, semantic digest 207 may be added to the fashion. Analog form 203 of course includes semantic 

original digital representation; in others, it may be added just information 205 and representation 207 of semantic digest 

prior to production of the analog form. Any representation of 411. Here, the bar code is used, but representation 207 could 

semantic digest 207 which is delectable from analog form 25 equally be part of a watermark, as in analog form 303. 

203 may be employed; in technique 201, semantic digest Components 405, 409, and 413 may be implemented as 

207 is a visible bar code. Of course, semantic digest 207 may programs executed on a digital computer system; analog 

include additional information; for example, it may be form producer 415 may be any device which can output an 

encrypted as described above and semantic digest 207 may analog form. 

include an identifier for the user whose public key is 30 Authenticating an Analog Form that Has a Semantic Digest 

required to decrypt semantic digest 207. In such a case, FIG. 5 shows a system 501 for authenticating an analog 

semantic digest 207 is a digital signature that persists in the form 503 that has a semantic digest 207. Analog form 503 

analog form. is first provided to semantic digest reader 505 and to 

With watermarking, the semantic digest can be invisibly semantics reader 505. Semantic digest reader 505 reads 

added to the analog form. This is shown in FIG. 3. In 35 semantic digest 207; if semantic digest 207 is a bar code, 

technique 301, analog form 303 again includes semantic semantic digest reader 505 is a bar code reader; if semantic 

information 305. Prior to producing analog form 303, the digest 207 is included in a digital watermark, semantic 

semantic information in the digital representation from digest reader 505 is a digital watermark reader which 

which analog form 303 is produced is used as described receives its input from a scanner. If semantic digest 505 must 

above to produce semantic digest 207; this time, however, 40 be decrypted, semantic digest reader 505 will do that as well, 

semantic digest 207 is incorporated into watermark 307, In some cases, that may require sending the encrypted 

which is added to the digital representation before the analog semantic digest to a remote location that has the proper key. 

form is produced from the digital representation and which, Semantics reader 507 reads semantic information 305. If 

like the bar code of FIG. 2, survives production of the analog analog form 503 is a document, semantics reader 507 is a 

form. A watermark reader can read watermark 307 from a 45 scanner which provides its output to OCR software. With 

digital image made by scanning analog form 303, and can other images, the scanner provides its output to whatever 

thereby recover semantic digest 207 from watermark 307. image analysis software is required to analyze the features of 

As was the case with the visible semantic digest, the the image that make up semantic information 305. If analog 

semantic digest in watermark 307 may be encrypted and form 503 is audio, the audio will be input to audio analysis 

may also function as a digital signature. 50 software. Once the semantics information has been reduced 

Adding a Semantic Digest to an Analog Form: FIG. 4 to semantics data 509, it is provided to semantic digest 

FIG. 4 shows a system 401 for adding a semantic digest maker 511, which makes a new semantic digest 513 out of 

to an analog form 203. The process begins with digital the information. To do so, it uses the same technique that 

representation 403, whose contents include semantic infor- was used to make old semantic digest 515. Comparator 517 

mation 205. Digital representation 403 is received by 55 then compares old semantic digest 515 with new semantic 

semantics reader 405, which reads semantic information 205 digest 513, if the digests match, comparison result 519 

from digital representation 403. Semantics reader 405*s indicates that analog form 203 is authentic; if they do not, 

operation will depend on the form of the semantic informa- result 519 indicates that they are not authentic. What 

tion. For example, if digital representation 403 represents a "match" means in this context will be explained in more 

document, the form of the semantic information will depend 60 detail below, 

on how the document is represented. If it is represented as "Matching" Semantic Digests 

a bit-map image, the semantic information will be images of With the digests that are normally used to authenticate 

alphanumeric characters in the bit map; if it is represented digital representations, exact matches between the old and 

using one of the many representations of documents that new digests are required. One reason for this is that in most 

express alphanumeric characters as codes, the semantic 65 digital contexts, "approximately correct" data is useless; 

information will be the codes for the alphanumeric charac- another is that the one-way hashes normally used for digests 

ters. In the first case, semantics reader 405 will be an optical are "cryptographic", that is, the value of the digest reveals 
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nothing about the value from which it was made by the hash 
function, or in more practical terms, a change of a single bit 
in the digital representation may result in a large change in 
the value produced by the hash function. Since that is the 
case, the only comparison that can be made between digests 
is one of equality. 

In the context of authenticating analog forms, the require- 
ment that digests be equal causes difficulties. The reason for 
this is that reading semantic information from an analog 
form is an error-prone operation. For example, after many 
years of effort, OCR technology has gotten to the point 
where it can in general recognize characters with 98% 
accuracy when it begins with a clean copy of a document 
that is simply formatted and uses a reasonable type font. 
Such an error rate is perfectly adequate for many purposes; 
but for semantic information of any size, a new digest will 
almost never be equal to the old digest when the new digest 
is made from semantics data that is 98% the same as the 
semantics data that was used to make the old semantic 
digest. On the other hand, if the semantics data obtained 
from the analog form is 98% the same as the semantics data 
obtained from the digital representation, there is a very high 
probability that the analog form is in fact an authentic copy 
of the digital representation. 
Precise Matches 

Of course, if the semantic information is limited in size 
and tightly constrained, it may be possible to require that the 
digests be exactly equal. For example, many errors can be 
eliminated if what is being read is specific fields, for 
example in a check or identification card, and the OCR 
equipment is programmed to take the nature of the field's 
contents into account. For example, if a field contains only 
numeric characters, the OCR equipment can be programmed 
to treat the letters o and O as the number 0 and the letters l,i, 
or I as the number 1. Moreover, if a match fails and the 
semantic information contains a character that is easily 
confused by the OCR equipment, the character may be 
replaced by one of the characters with which it is confused, 
the digest may be recomputed, and the match may again be 
attempted with the recomputed digest. 
Fuzzy Matches 

Where the semantic information is not tightly constrained, 
the digests must be made in such a fashion that closely- 
similar semantic information produces closely-similar 
digests. When that is the case, matching becomes a matter of 
determining whether the difference between the digests is 
within a threshold value, not of determining whether they 
are equal. A paper by Marc Schneider and Shih-Fu Chang, 
"A Robust Content Based Digital Signature for Image 
Authentication", in: Proceedings of the 1996 International 
Conference on Image Processing, presents some techniques 
for dealing with related difficulties in the area of digital 
imaging. There, the problems are not caused by loss of 
information when a digital representation is used to make an 
analog form and by mistakes made in reading analog forms, 
but rather by "lossy" compression of images, that is, com- 
pression using techniques which result in the loss of infor- 
mation. Because the lost information is missing from the 
compressed digital representation, a digest made using cryp- 
tographic techniques from the compressed digital represen- 
tation will not be equal to one made from the digital 
representation prior to compression, even though the com- 
pressed and uncompressed representations contain the same 
semantic information. Speaking generally, the techniques 
presented in the Schneider paper deal with this problem by 
calculating the digest value from characteristics of the image 
that are not affected by compression, such as the spatial 
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location of its features. Where there are sequences of 
images, the digest value is calculated using the order of the 
images in the sequences. 

Analogous approaches may be used to compute the 
5 semantic digest used to authenticate an analog form. For 
example, a semantic digest for a document can be computed 
like this: 

1. Set the current length of a digest string that will hold 
the semantic digest to "0", 
io 2. Starting with the first alphanumeric character in the 
document, perform the following steps until there are 
no more characters in the document: 

a. Select a next group of characters, 

b. For the selected group, 

35 i. replace characters in the group such as O,0,o; 

I,i,l,l, or c,e that cause large numbers of OCR 
errors with a "don't care" character; 
ii. make a hash value from the characters in the 
group; 

20 in\ append the hash value to the semantic digest 

string; 

c. return to step (a). 

3. When there are no more characters in the document, 
make the semantic digest from the digest string. 

25 When computed in this fashion, the sequence of values in 
the semantic digest string reflects the order of the characters 
in each of the sequences used to compute the digest. If the 
sequence of values in the new semantic digest that is 
computed from the analog form has a high percentage of 

30 matches with the sequence of values in the old semantic 
digest, there is a high probability that the documents contain 
the same semantic information. 
Applications of Authentication with Analog Forms 

One area of application is authenticating written docu- 

35 ments generally. To the extent that the document is of any 
length and the digest is computed from a significant amount 
of the contents, the digest will have to be computed in a 
fashion which allows fuzzy matching. If the digest is com- 
puted from closely-constrained fields of the document, exact 

40 matching may be employed. 

Another area of application is authenticating financial 
documents such as electronic cash, electronic checks, and 
bank cards. Here, the fields from which the digest is com- 
puted are tightly constrained and an exact match may be 

45 required for security. In all of these applications, the digest 
or even the semantic information itself would be encrypted 
as described above to produce a digital signature. 
Universal Paper & Digital Cash 

Digital cash is at present a purely electronic medium of 

50 payment. A given item of digital cash consists of a unique 
serial number and a digital signature. Authentication using 
semantic information permits digital cash to be printed as 
digital paper cash. The paper cash is printed from an 
electronic image which has a background image, a serial 

55 number, and a money amount. The serial number and the 
money amount are the semantic information. The serial 
number and the money amount are used to make a digital 
signature and the digital signature is embedded as an elec- 
tronic watermark into the background image. The paper cash 

60 can be printed by any machine which needs to dispense 
money. Thus, an ATM, can dispense digital paper cash 
instead of paper money. Similarly, a vending machine can 
make change with digital paper cash and a merchant can do 
the same. The digital paper cash can be used in the same way 

65 as paper money. When a merchant (or a vending machine) 
receives the digital paper cash in payment, he or she uses a 
special scanner (including OCR technology and a watermark 
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reader) to detect the watermark (i.e. the serial number and representations, or it may be defined specifically for a given 
money amount) from the printed image, and send them to digital representation. Code 611 may of course also use the 
the bank for verification in the same fashion as is presently other information in watermark information 603 as data, 
done with credit cards. Watermark information 603 and digital representation 613 
Digital Checks 5 are input into watermark maker 615, which outputs digital 
Digital checks can be made using the same techniques as representation 617, which is digital representation 613 
are used for digital paper cash. The digital check includes a modified to include watermark 619 made from watermark 
background image, an identifier for the bank account, an information 603. Since watermark information 603 includes 
amount to be paid, and the name of the payer. The payer's code 611, watermark 619 is an active watermark, 
private key is used to make a digital signature from at least 10 FIG. 11 shows a preferred embodiment of access infer- 
tile identification of the bank and the amount to be paid, and mation 607. It includes fields as follows: 
the digital signature is embedded as an electronic watermark an 8-bit permission (P) field which indicates the kind of 
in the background image. Writing a digital check is a access the user may have: among the kinds are access 
three-step process: enter the amount, produce the digital which permits display, access which permits storing a 
signature from the bank account number and the amount is local copy, and access which permits printing, 
using the payer's private key, and embed the digital signa- a four . bit sensitivity field whose value indicates the 
ture into the background image. The bank verifies the check degree of sensitivity of the contents of the digital 
by detecting the watermark from the digital check, decrypt- representation - 

ing the digital signature with the payer's public key and 32-bit allowed location field which contains tbe IP address 

comparing the bank account number and the amount from 20 at whjch , he ^ { representation ^ permitted to be 

the image with the bank account number and the amount on located' 

the face of the check. A digital check can be used in either , . , « . , . • * j e 

electronic form or paper form. In the latter case, a scanner 32-bit allowed period field which contains a period of 

(including OCR technology and watermark reader) is " me for wmcb of the dl S ,tal re P resentatl0n has 

needed to read the watermark from the paper check. 25 permi e » j 

Authentication of Identification Cards . F1 °' 7*™ example ^of a program which might be found 

The techniques described above for authenticating digital mcod ? 61t am ! 01 15 wn "f » ,n *e Java program- 
paper cash or digital checks can be used with identification rmng language. It is then compiled into Java bytecodes 
cards, including bankcards. Hie card number or other iden- which f K A f?^*?. * f b y te ^ es 
tification information appears on the face of the card, is 30 are included m the digUal watermark When program 701 is 
encrypted into a digital signature, and is embedded as a ™ ted ' ame! fg e md.cat.ng that d.g.tal representation 
digital watermark inThe background image of the bankcard. 617 «V U " W « the actlve watenna * has been displayed is 
Tie encryption can be done with the private key of the »* . v,a the Internet to a system that has been set up to 
institution that issues the card. The merchant uses use a momtor the d f la ? 'eP™Uon 617 perhaps for 

* a* 4 *u a' i r n „ n ,j n „ m u 0 « the purpose of compuhng license tees. Line 703 ot the code 

scanner to detect the digital signature (i.e. card number or 35 v v t , r . 

other ID) from the card/and compare the signature with the ^ a socket * by means of which a datagram may be sent 

authentication stored inside the card. Tins technique can of t0 the monitoring system. Line 709 of the code finds the 

course be combined with conventional authentication tech- current Intcrnet address a of the monit °™S svstem which IS 

niques such as the holographic logo. specified at 705 by the name &syscop.crg.edu&. Line 

Active Watermarks: FIGS. 6-8 40 17 1 5 makes a new datagram packet for the message; it 

Heretofore, digital watermarks have been nothing more includes the message content, &XYZ Displayed & and the 
than labels. They have typically contained information such internet address a. Line 1719, finally, uses the send operation 
as identifiers for the owner and creator of the digital repre- associated with the socket s to send the message, which the 
sentation and access control information, for example, Internet will deliver to the destination specified by a. 
whether the digital representation may be copied or 45 FIG. 8 shows a system 801 for executing the code in 
changed. Any kind of information can, however be placed in active watermark 619. Digital representation 617 containing 
a digital watermark. If the information in the watermark active watermark 619 is input to watermark reader 803, 
describes an action to be taken, the watermark becomes which extracts watermark information 603 from active 
active, and the digital representation that contains the active watermark 619. Info 603 includes code 611, which water- 
watermark becomes active as well. This is the reverse of the 50 mark reader 803 provides to code interpreter 805, Code 
usual practice of encapsulating a digital representation in a interpreter 805 interprets code 611 to provide instructions 
program, as is done for example with Microsoft Active which are executable by the computer system upon which 
Documents. Since digital watermarks are used in digital code interpreter 805 is running. In some embodiments, code 
systems, the simplest way to make a watermark active is to interpreter is an interpreter provided by the computer system 
include program code in it which can be executed by the 55 for a standard language such as Java; in others, interpreter 
computer system upon which the digital representation is 805 may be provided as a component of watermark reader 
currently resident. From the point of view of function, the 803. In such embodiments, code 611 may be written in a 
code may be in any language for which the computer system language specifically designed for active watermarks, 
can execute code. Practically, however, the code is best An active watermark 619 can cause the computer system 
written in a language such as JavaTM or Perl for which most 60 in which the active watermark is read to perform any action 
modern computer systems have interpreters. which can be described by the code contained in the active 

FIG. 6 is an overview of a system 601 for making an watermark. The only limitations are those imposed by the 

active watermark 619. The watermark is made from water- fact that the code is part of a watermark. One of these 

mark information 603, which contains owner information limitations is code size: code contained in a watermark must 

605, access information 607, and owner-defined information 65 necessarily be relatively short; this limitation can be allevi- 

609 as before, but additionally contains code 611. Code 611 ated by compressing the code using a "non-lossy" compres- 

may be standard code for a given class of digital sion technique, that is, one which does not result in the loss 
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of information. Another of the limitations is that damage to 
the watermark may result in damage to the code, 
consequently, active watermarks may not work well in 
situations where the digital representation 617 is involved in 
"lossy" manipulations, i.e., manipulations that cause loss of 5 
information in digital representation 617. Examples of such 
lossy manipulations are editing the digital representation, 
lossy translation of the digital representation from one 
format into another, lossy compression of the digital 
representation, and producing a new digital representation ]Q 
from an analog form made from an old digital representation 
(as would be the case, for example, if the code were obtained 
by reading the watermark from a paper copy of a document). 

Of course, techniques like those discussed above with 
reference to exact matches of digests can be applied to 35 
recover code from a damaged watermark or from an analog 
form, and to the extent that such techniques are successful, 
active watermarks can be used even where lossy manipula- 
tions have taken place. For example, the watermark of an 
analog form may contain not only authentication 2Q 
information, but also code. If a copy machine contained a 
watermark reader and an interpreter for the code used in the 
active watermark, the active watermark could be used, for 
instance, to prevent the copy machine from copying the 
analog form. 25 

Among the things that can be done with active water- 
marks are the following: 

Customizing the manner in which the digital representa- 
tion containing the watermark is treated. Code 611 may 
differ for classes of digital representations, or may even 30 
be particular to a single digital representation, 
Having a digital representation send a message whenever 
it is displayed, copied, printed, or edited; for example, 
whenever a document with an active watermark stored 
on a Web server is downloaded from the server, the 35 
active watermark can cause a message containing bill- 
ing information to be sent to a billing server. 
Having the digital representation obtain locally-available 
information, which will then govern the behavior and 
usage of the digital representation; 40 
Having a digital representation take protective action 
when a user tries to do something with it that is not 
permitted by access information 603; the protective 
action can range from a warning through sending a 
message or blocking the intended action to destroying 45 
the digital representation that contains the watermark. 
Watermark Agents 

Digital representations pose special problems for their 
owners because, like all digital data, they can be easily 
copied and distributed across a network. These properties of 50 
digital data, however, also make it possible to automate 
monitoring of the distribution and use of watermarked 
digital representations. One way to do this is the watermark 
spider. As mentioned in the Description of the Prior Art, the 
watermark spider follows URLs to Web pages, which it 55 
retrieves and inspects for watermarks. If it finds one that is 
of interest, it reports its findings to a monitoring program. 
There are two problems with the watermark spider: the first 
is that it is limited to digital representations which are 
accessible by URLs that are available to the public. Thus, the 60 
watermark spider would not be able to locate a copy of a 
digital representation on a WWW client, as opposed to on a 
WWW server. The other problem is that the spider must 
fetch each digital representation to be examined across the 
network. Since digital representations are often large, the 65 
need to do this adds substantially to the volume of network 
traffic. 



Both of these problems can be solved by means of a 
network watermark agent, that is, a watermark monitor 
which uses the network to move from system to system 
where digital representations of interest might be stored. At 
each system, the watermark agent examines the system's file 
system for digital representations that have watermarks of 
interest. If the watermark agent finds such a watermark, it 
may send a message with its findings via the network to a 
monitoring program. The watermark agent is thus able to 
monitor digital representations that are not available via 
public URLs and uses network bandwidth only relatively 
rarely and only to send messages that are small in compari- 
son with digital representations. In the following, the cre- 
ation of a watermark agent and its behavior in a system will 
both be explained in detail. 
Creating a Watermark Agent: FIG. 9 

FIG. 9 shows a watermark monitoring system 901 which 
creates and dispatches a watermark agent 925 across a 
network 103 and responds to messages from the watermark 
agent. Watermark agent 925 is a program which is able to 
send itself from one node to another in network 103. In each 
node, it searches for watermarked documents and sends 
messages 935 containing its findings to monitoring system 
901, where message handler 920 deals with the message, 
often by adding information to management data base 903. 

Continuing in more detail, agent 925 has two main parts: 
agent code 927, which is executed when agent 925 reaches 
a node, and agent data 929, which contains information used, 
by agent 925 in executing the code and in moving to the next 
node. At a minimum, agent code 927 will include code 
which searches the node for files that may contain 
watermarks, code that makes and sends any necessary 
messages to monitoring system 901, code that clones agent 
925, and code that sends the clone on to the next node. As 
with the code in active watermarks, code 927 may be written 
in any language which can be executed in a node; either 
standard languages such as Java or a specialized watermark 
agent language may be used. 

FIG. 12 provides an example written in the Java language 
of code 1201 that a watermarking agent 925 might execute. 
Code 1201 searches the file system of the network node at 
which agent 925 is presently located for images files, checks 
each image file for a watermark, if it finds a watermark, it 
performs the action required by the watermark and the node, 
and makes a message containing a list of the actions it 
performed. 

Continuing in more detail, code 1201 has two main 
sections, initialization 1203 and checking loop 1213. In 
initialization 1203, the first step is to instantiate a file filter 
to filter the files in the node's file system (1205). Then a 
function of the filter which locates image files is used to 
make a list filenames of the names of the image files in the 
file system (1207). Thereupon, information about the envi- 
ronment of the node that the agent needs to check water- 
marks is retrieved and placed in a variable env (1209); 
finally, a data structure called results is created to hold the 
results of the watermark checks 1211. 

In loop 1213, each file in filenames is examined in turn for 
a watermark (1215); if one is found, the actions indicated at 
1217 are performed, first, the contents of the watermark are 
compared with the environment information to obtain a 
result called match (1219). Then match is passed to a 
function which takes an action as determined by the value of 
match and returns a value result which represents the result 
of the action (1221); finally, result is added to the data 
structure results (1223) then, at 1225, results is returned. 
Depending on how the watermark agent is being used, 
results can then be sent in a message to monitoring system 
901. 
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Continuing in more detail with agent data 929, agent data 
929 includes a map 931, digital representation description 
933, keys 934, and parameters 921. Map 931 is a list of 
addresses in network 103. Each address specifies an entity in 
network 103 that can provide an environment in which agent 5 
925 can operate. The address may for example be an E-mail 
address or an IP address. Digital representation description 

933 may be any information that describes the digital 
representations the agent is looking for. There may be a filter 
for the file names and there may also be identification 
information from the watermark. For example, if the files to 
be examined are .bmp files, the filter might specify *.bmp, 
indicating that all files with the .bmp suffix are to be 
examined. If a watermark key is needed to read the 
watermark, keys 934 will contain that key and if the mes- 
sages sent to monitor system 901 are to be encrypted, keys 15 

934 will contain the key to be used in encrypting the 
messages. Any available technique may be used to keep the 
keys secure. In a preferred embodiment, the parameters 
include 

the email address for messages sent by the agent; 20 
whether to report on files to which agent 925 had no 
access; 

date of last monitoring and whether to check only files 

updated since that date; 
whether to execute code 611 in an active watermark 619; 25 

and 

termination conditions for agent 925. 

Agent 925 is produced by agent generator 923, which can 
be implemented as a component of digital representation 
manager 131. Agent generator 923 makes agent 925 from 30 
information in management data base 903 and agent param- 
eters 921, which here are shown being provided interac- 
tively by a user of monitoring system 901, but may also be 
stored in management data base 903. The information in 
management data base 903 includes agent template 905(i), 35 
which is one of a number of templates that are used together 
with parameters 921 and other information in management 
data base 903 to generate agent code 927 for different kinds 
of agents 925. Suspicious sites 907 is a list of network 
locations which might be worth examining. One source of 
information for sites that should be on the list of suspicious 
sites 907 is of course messages received from previously- 
dispatched agents. Network information 909 is information 
about the network. Suspicious sites 907 and network infor- 
mation 909 are used together to make map 931 in agent 925. 
Digital representation information 911, finally, contains 45 
information about the digital representations that the agent 
will be looking for. The information is used to make DR 
Description 933. Information 911(z) for a given digital 
representation or group of digital representations may 
include a watermark key 913 for the digital representation's 50 
watermark and information from the watermark including 
owner ID 915, user ID 917, and permitted use information 
919. User ID 917 is an identification for the user to whom 
the digital representation was downloaded. Once agent 905 
has thus been created by monitor system 901, agent 925 55 
clones itself, makes the clone into the kind of message 
required for the first entity specified in map 931, and sends 
the message to the first entity. Thereupon, agent 925 termi- 
nates itself 

Watermark Agents in Network Nodes: FIG. 10 eo 

FIG. 10 shows those components of a network node 1001 
which are involved in the monitoring of the node by a 
watermark agent 925 The components include: 

agent engine 1003, which provides the environment in 
which agent 925 executes its code and which is the 65 
entity to which the message containing agent 925 is 
addressed; 
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file storage 1031, which contains the digital representa- 
tions 1023 that are of interest to agent 925; 

file system 1029, which makes the digital representations 
1023 accessible as files, 

watermark reader 1019, which reads the watermarks; and 

code interpreter 1011, which interprets code in agent 925 
and may also interpret the code in active watermarks, 
if that code is written in the same language as the code 
used in agent 925. 

SC 1035 is an optional secure coprocessor whose func- 
tions will be explained in more detail in the discussion of 
security. 

Operation of components 1001 is as follows: When the 
message containing agent 925 arrives in agent engine 1003 
from network 103, agent engine 1003 extracts agent 925 
from the message and, at a convenient time, uses code 
interpreter 1011 to begin executing its code. What the code 
does is of course arbitrary. Typically, it will do the following: 

1 . Send a message to system 901 indicating its arrival in 
the node; 

2. Obtain the file filter from DRDESC 993 and give it to 
spider 1009 to make a list of files that match the filter; 

3. For each file on the list, do the following: 

a. use spider 1009 to get the file ID for the file, 

b. give file ID 1021 to watermark reader 1019, which 
uses the watermark key from keys 934 to read the 
watermark, if any; 

c. receive the watermark content 1017, 

d. process watermark content 1017 as specified in code 
927. Actions might include sending a message to 
system 901 or passing the code and data 1015 from 
an active watermark to code interpreter 1011 for 
execution and receiving data 1013 in return, 

4. When all of the files have been processed, 

a. sending a message to monitor system 901 with 
summary information about the results of the visit 
and the next node to be visited; 

b. making a clone of agent 925 and sending the clone 
to the next address specified in map 931; and 

c. terminating agent 925. 

As previously indicated, what a watermark agent can do 
is essentially arbitrary. If the documents being dealt with by 
the watermark agent have active watermarks, there are any 
number of ways of dividing the work of processing of 
documents of interest between the code in the watermark 
agent and the code in the active watermark. For instance, in 
the example above, step 3(d) above could consist simply of 
executing the code in the document's active watermark. 

The actions performed in step3(d) will typically be per- 
formed when the information in the watermark does not 
match the time or place where agent 925 found the file or the 
time and/or place are inappropriate for the file's access 
privileges. The action may be one of a pre-defined set 
specified by parameters in parameters 921, it may be one 
defined by agent 925 's code 927, or it may be one defined 
by an active watermark. Among the predefined actions are: 

1. Destroy the file if the file's sensitivity level is very high; 

2. Remove the file to a safe place if the sensitivity level 
is medium, 

3. If the sensitivity level is low, 

a. Warn the local administrator or webmaster of the 
violation if the sensitivity is low; 

b. Warn the recipient of the violation if the sensitivity 
is low; or 

c. Send a message to the file's owner reporting the 
violation if the sensitivity is low; 
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4. If the sensitivity level is very low, send a message to of agent engine 1003. If the message were not properly 

monitor 901 without disturbing the local host and local answered, the digital representation manager might require 

administrator. that the node download and install agent engine 1003 before 

Before going on to the next destination, watermark agent proceeding further with the transaction. 

925 may, wait for a message from monitor 901 containing 5 The remainder of these problems are termed "malicious 

information about the next destination; the information may node problems". They can be solved by standard crypto- 

include: graphic techniques, as described in Schneier, supra. For 

The time of the last visit by an agent to the destination; example, the digital representation manager and each agent 

Information about the destination, for example detailed en S ine 1003 have a ^ key-private key pair In that 

information about the digital representations to be 10 ^ e ^ etwork information 909 would l " clude [ he public ly 

examined there agent engine 1003 at a given node and the public keys 

Nontraveling Watermark jfents f ° T the , agent engines 1003 in the nodes to be visited would 

An important difference between a watermark agent and be mcluded in ma P 93L messa 6 e ""L^ tne dl & til 

a watermark spider is that the watermark agent interacts with re P resen i t ^° n ma ° a S er 01 ^ an . a 8 ent 925 t0 . ™ a £ nt 

the document in the system where the document is being 15 engine 1003 can be encrypted using agent engine 1003 s 

stored or processed, and can thus perform far more functions P ubhc ke y and ^message «"t by an agent engine 1003 or 

than a watermark spider can. A further consequence of this ana g en l t 925 to f dl 8 1,al ^presentation manager can be sent 

difference is that a watermark agent need not travel, but can usl u n S *e digital representation manager s pubhc key. The 

simply be incorporated as a permanent component of a P ubl,c kev for the digital representation manager can of 

system. For example, a copier could include a watermark 20 c ° urse be included in agent 925 s keys 934 Authentication 

agent that read the watermarks of paper documents being of messages can be done using standard digital signature 

copied and prevented the copier from copying a document '^.ques; for f™? 1 *' a 8? nt , data 929 mi 8 nt mclude f a 

when its watermark indicated that the document was not to d ig«al signature from the digital representation manager for 

be copied. An important application of such a non-traveling a 8 ent 925 > messages from the digital representation manager 

watermark agent would be to prevent the copying of paper 25 lo a S ent f n 8 ine 1003 can lnclude ' he dl g' tal ^presentation 

digital cash manager s digital signature, and messages from agent 

Of course, if the copier had access to a network, even the en § ine 1003 can include a 8 ent en 8 ine 10ft3 ' s si ^ a ' 

"non-traveling" watermark agent could at least travel via the tur 5; , , , ... 
network to the copier, and the network would provide a »j the watermarks are made usmg encryption techniques 
convenient way of updating the copier's watermark agent. 30 " * sa * ed m * Koch u and *• , Zha °' Towards Robusl and 
"Non-traveling" watermark agents could of course be dis- " ldden Ima S' Copyright Labeling , supra, the agent must 
tributed in a similar fashion to any system accessible via the have a way of decrypting the watermark. Depending on the 
network situation, the watermark may be encrypted with the water- 
Security Considerations mark agent's public key and authenticated with a digital 

In some cases, for example in private military or business 35 S1 & nature m u the same fa f hl0n a * other messa f s 

networks or systems, agent 925 may not operate in a hostile f S me or the watermark may have its own key 913. In the 

environment, and monitor 901 and agent engine 1003 may former case ' < ne ' th f. watermark agent s private key must be 

even be implemented as integral parts of the operating P rotecte ^ md in the latte u r ' ^termark key 913 must be 

system. In most cases, however, agent 925 will be operating protected, since access to the key would permit those intent 

in an environment which is hostile in at least four respects: 40 ° n stea hn S dl & ltal . representations to remove or alter the 

, , i . , Mm . . ir . . digital representation s watermark. While agent 925 is in 

The node to which agent 925 sends iteclf is properly ^ watermark k 913 can be tected b encryption in 

suspicious of messages from outside that contain code ^ same fashioQ ag ^ fest of ^ information in , 925; 

to be executed on the node; Qnce 925 hag beeQ decryptedj watermark key 913 and 

to the extent that users on the node have violated the 45 agent eagin6 im > s private key must be p rotec ted in the 

conditions under which they received a digital node current i y be i ng visited by agent 925. Agent engine 

representation, they will want to hide their behavior 1003 > s private key must fo rt h er be protected to prevent a 

and/or disable agent 925; user of the node current i y be in g visited by agent 925 from 

users on the node may want access to the keys and other us ing the private key to decrypt messages addressed to agent 

data carried by agent 925; and 50 engine 1003 or append agent 1003 , s digital signature, 

other users of network 103 may be interested in the One way of solving these key protection problems is a 

content of the messages being exchanged between secure coprocessor, as described in J. D. Tyger and Bennet 

agent 925 and monitor 901. Lee, Secure Coprocessors in Electronic Commerce 

The first of these problems is the "malicious agent prob- Applications, FIRST USENIX WORKSHOP ON ELEC- 

lem". It is general to systems that download and execute 55 TRONIC COMMERCE, JULY 1995. As shown at 1033, a 

code, and the same solutions that are used in those cases can secure coprocessor includes secure storage 1035 and a 

be applied to agent engine 1003 and agent 925. For example, secure processor 1045. Secure storage 1035 may only be 

if the watermark agent's code is written in Java, the system accessed via secure processor 1045, and secure coprocessor 

on which it is run will have whatever protections are 1033 is built in such a fashion that any attempt to access the 

provided by the Java interpreter. If managers of nodes are 60 information in secure coprocessor 1033 other than via secure 

reasonably certain that agent engine 1003 and agents 925 processor 1045 results in the destruction of the information, 

will not do any damage to the nodes, they can be made to Secure coprocessor 1033 is able to write information to and 

accept engine 1003 and agents 925 simply as a condition of read information from secure storage 1035 and also does 

downloading digital representations. For example, the trans- encryption and decryption and makes and verifies digital 

action by which a digital representation manager downloads 65 signatures. These operations may be done entirely by 

a digital representation to a node might include a message to executing code stored in secure storage 1035 or by means of 

agent engine 1003 confirming the existence and operability a combination of code and specialized hardware devices, as 
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shown at 1047 and 1049. The keys used in encryption, rights to the document or moving the document to a safe 

decryption, and in making digital signatures and verifying location to immediate destruction of the out-of-place docu- 

them are stored in secure storage 1035. Shown in FIG. 10 are ment. Again, the agent that does this need not travel, but may 

WMkey 913 for the watermark, monitor public key 1039, simply be a permanent component of the file system, 

agent engine public key 1041, and agent engine private key 5 Watermark agents, finally, can be used to find lost docu- 

1043. In the case of the public keys, storage in secure storage ments in military or business file systems or networks. If 

1035 is simply a matter of convenience, and secure proces- each document has a unique identifier associated with it and 

sor 1045 may provide access to the public keys in response tbat id ? ntm ? r * ° n lhe one ha f d ke P l m a databa f and ° n 

to requests from components of node 1001, in the case of < he other hand incorporated into a watermark m the 

WMkey 913 and agent engine 1003's private key 1043, the 10 docume f> a watermark agent can simply be gxven the 

j j j i ft1 , , I**- j i universal identifier and sent to search the file system or 

decrypted keys 913 and 1043 are used only within secure netWQrk fof {h& document 0nce the agcnt has [on J it> it can 

processor 1033. report its location to whomever sent the agent out. 

In the context of system 1001, when a message encrypted Conclusion 

with agent engine 1003's public key 1041 arrives in agent foregoing Detailed Description has disclosed to those 

engine 1003, agent engine 1003 uses secure processor 1033 15 skilled in the relevant arts how to make and use documents 

to decrypt the message; if the message contains an agent with authentication that withstands conversion between an 

925, agent engine 1003 also uses secure processor 1033 to analog form and a digital representation of the document, 

verify that agent 925 's digital signature is from the digital how to make and use digital representations with active 

representation manager and to decrypt WMkey 913. The watermarks, and how to make and use watermark agents, 

decrypted key is not returned to agent engine 1003, but is 20 including mobile watermark agents and has further disclosed 

stored in secure storage 1035. SWM reader 1019 then uses the best mode presently known to the inventors for making 

secure coprocessor 1033 to decrypt the watermark in the such authentications, making active watermarks, and mak- 

digital representation currently being checked by agent 925. ing watermark agents. The disclosed techniques are exceed- 

Applications Using Watermark Agents ingly general and may be implemented in many different 

Since a watermark agent is programmed, it can do literally 25 wavs for man y different purposes. For example, the authen- 

anything. The flexibility of watermark agents is increased tication techniques may be based on any kind of semantic 

when their use is combined with that of active watermarks. information and there are many ways of deriving the authen- 

One set of applications for watermark agents is monitoring tication information from the semantic information, placing 

the use of copyrighted digital representations for the copy- the authentication information in the digital representation 

right owner or a licensing agency. A copyright owner or 30 or me analo S form > and comparing the authentication infor- 

licensing agency, for example, may use watermark agents to mation - Similarly, the program code for an active watermark 

locate unlicensed copies of digital representations or to ma y be m ^ programming language, may be in 

periodically monitor the use of licensed copies. A document source or ob J ect form ' and mav > when executed, perform 

with an active watermark could increment a usage count arbitrarv operations. Watermark agents too, may perform 

maintained in agent engine 1003 for a node each time it was 35 arbitrarv actions and em P lo y vanous techniques for sending 

printed and agent 925 could read the count on its visit to the messages and traveling from node to node in a network. The 

node, report the current count value back to management watermark agents can of course perform authentication 

database 903 and reset the counter information and can execute code from active watermarks. 

Another set of applications is monitoring the use of digital Since the techniques are so general and may be imple- 

representations to avoid liability for infringement. For 40 mented in any number of ways, the Detailed Description is 

example, a corporation might want to be sure that it has no t0 be regarded as being in all respects exemplary and not 

unauthorized digital representations in its network and that restrictive, and the breadth of the invention disclosed herein 

the authorized ones are being used in accordance with their * t0 * determined not from the Detailed Description, but 

license terms. The agent can monitor the use of the digital rather from the claims as interpreted with the full breadth 

representations in the corporate network in the same fashion 45 permitted by the patent laws, 

as it does for the licensing agency. In this instance, the wbat ^ chimed is: 

monitoring might even include destroying illegal copies. 1- Improved digital watermarking apparatus of the type 
Yet another set of applications is preventing unauthorized that receives ^formation and makes a digital watermark in 
copying, scanning, or printing. This can be done by means a digital representation, the watermark containing the infor- 
of"nontraveling" watermark agents on servers and clients in 50 matl0n and the improvement comprising: 
the network or even by means of "nontraveling" watermark including program code in the received information, the 
agents built into devices such as copiers, scanners, or program code specifying an action to be performed in 
printers. For example, if a "No copy" watermark is embed- response to performance of an operation on the digital 
ded in currency and a photocopier has an agent that looks for representation by a program of which the digital rep- 
such a watermark and inhibits copying when it finds the 55 resentation is not a component, 
watermark, the photocopier will not make copies of cur- whereby performance of the operation causes the program 



rency. 



code to be executed. 



Watermark agents can also be used to enforce military or 2 - The improved digital watermarking apparatus set forth 

corporate document security rules. In such an application, m claim 1 wherein: 

the document's security classification would be embedded in 60 the digital watermark does not affect the ability of pro- 
it as a watermark and the watermark agent would search the grams that process the digital representation to do so, 
military or corporate file systems and networks for docu- whereby no modification of such programs is required, 
ments that were not being dealt with as required by their 3. The improved digital watermarking apparatus set forth 
security classification. Examples would be documents that in claim 1 wherein: 
were in the wrong place or had been kept longer than a 65 the digital watermark is encrypted, 
predetermined period. Actions taken by the agent can range 4. The improved digital watermarking apparatus set forth 
from reports and warnings through changing the access in claim 3 wherein: 
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the digital watermark is associated with authentication 
information. 

5. The improved digital watermarking apparatus set forth 
in claim 1 wherein the operation comprises one or more of: 

copying the digital representation, displaying or playing 5 
the digital representation, printing the digital 
representation, modifying the digital representation, 
modifying the digital representation's relationship to a 
file system, modifying access privileges associated 
with the digital representation, or further distributing 10 
the digital representation. 

6. The improved digital watermarking apparatus set forth 
in any one of claim 1 or 5 wherein the action comprises one 
or more of: 

sending a message, inhibiting the operation, or destroying 35 
the digital representation. 

7. The improved digital watermarking apparatus set forth 
in claim 6 wherein the action of sending a message com- 
prises one or more of: 

. .20 

sending a usage monitoring message to a usage monitor- 
ing system, sending a billing information message to a 
billing system, sending an informational message to a 
user for whom the operation is being performed, or 
sending a security message to a security system. 25 

8. A method of making a digital watermark comprising the 
steps of: 

receiving program code to be included in the digital 
watermark, the program code specifying an action to be 
performed in response to performance of an operation 30 
on a digital representation containing the watermark by 
a program of which the digital representation is not a 
component; and 

making the digital watermark with the program code. 

9. Improved digital watermark reading apparatus of the 35 
type which reads a digital watermark in a digital 
representation, the improvement comprising: 

a program code interpreter, the digital watermark reading 
apparatus using the program code interpreter to execute 
program code included in the digital watermark in 40 
response to an operation that is performed on the digital 
representation by a program of which the digital rep- 
resentation is not a component. 

10. The improved digital watermark reading apparatus set 
forth in claim 9 wherein: 45 

the digital watermark does not affect the ability of pro- 
grams that process the digital representation to do so, 
whereby no modification of such programs is required, 

11. The improved digital watermark reading apparatus set 
forth in claim 9 wherein: 

the digital watermark is encrypted. 

12. The improved digital watermark reading apparatus set 
forth in claim 9 wherein the operation comprises one or 
more of: 55 

copying the digital representation, displaying or playing 
the digital representation, printing the digital 
representation, modifying the digital representation, 
modifying the digital representation's relationship to a 
file system, modifying access privileges associated 60 
with the digital representation, or further distributing 
the digital representation. 

13. The improved digital watermark reading apparatus set 
forth in any one of claim 9 or 12 wherein the action 
comprises one or more of: $5 

sending a message, inhibiting the operation, or destroying 
the digital representation. 



50 



14. The improved digital watermark reading apparatus set 
forth in claim 13 wherein the action of sending a message 
comprises one or more of: 

sending a usage monitoring message to a usage monitor- 
ing system, sending a billing information message to a 
billing system, sending an informational message to a 
user for whom the operation is being performed, or 
sending a security message to a security system. 

15. A method of reading a digital watermark comprising 
the steps of: 

performing an operation on a digital representation con- 
taining a watermark, the operation being performed by 
a program of which the digital representation is not a 
component; and 

if the digital watermark contains program code, executing 
the program code in conjunction with the performance 
of the operation. 

16. Improved apparatus for performing an operation on a 
digital representation, the apparatus having the improvement 
comprising: 

a watermark reading apparatus that includes a program 
code interpreter for interpreting program code included 
in a digital watermark in the digital representation and 
the program code specifying an action to be performed 
if the operation is performed on the digital 
representation, the operation being performed in the 
apparatus by a program of which the digital represen- 
tation is not a component. 

17. The improved apparatus set forth in claim 16 wherein: 
the improved apparatus is a printing apparatus and the 

program code interpreter interprets the program code if 
the digital representation is printed thereby. 

18. The improved apparatus set forth in claim 16 wherein: 
the improved apparatus is a display apparatus and the 

program code interpreter interprets the program code if 
the digital representation is displayed thereby. 

19. The improved apparatus set forth in claim 16 wherein: 
the improved apparatus makes a copy of the digital 

representation and the program code interpreter inter- 
prets the program code if the digital representation is 
copied thereby. 

20. The improved apparatus set forth in claim 16 wherein: 
the improved apparatus distributes the digital representa- 
tion via a network and the program code interpreter 
interprets the program code on distribution of the 
digital representation. 

21. The improved apparatus set forth in claim 16 wherein: 
the improved apparatus is an executing editor for the 

digital representation and the program code interpreter 
interprets the program code if the digital representation 
is altered by the editor. 

22. The improved apparatus set forth in claim 16 wherein: 
the improved apparatus is an executing watermark agent 

and the program code interpreter interprets the program 
code if the watermark agent so indicates. 

23. The watermark agent set forth in claim 22 wherein: 
the digital representation resides in a node in a network; 

and the watermark agent 
travels via the network to the node. 

24. The improved apparatus set forth in any one of claims 
16 through 23 wherein: the 

improved apparatus is implemented in a browser. 

25. The improved apparatus set forth in claim 16 wherein: 
the digital representation is made from an analog form 

that includes the digital watermark. 
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26. The improved apparatus set forth in claim 25 wherein: 
the improved apparatus is implemented in a copier that 

makes a copy of the analog form. 

27. An improved digital watermark in a digital 
representation, the improvement comprising: 

a representation in the digital watermark of program code, 
the program code specifying an action to be performed 
in response to performance of an operation on the 
digital representation, the operation being performed 
by a program of which the digital representation is not 
a component. 

28. A method of receiving information about a digital 
representation that includes a watermark that contains pro- 
gram code which is executed when an operation is per- 
formed on the digital representation and if executed, causes 
a message to be sent, the operation being performed by a 
program of which the digital representation is not a com- 
ponent and the method comprising the steps of: 

distributing the digital representation; and 
receiving the message. 

29. The method set forth in claim 28 further comprising 
the step of: 

adding the watermark to the digital representation. 

30. A method of obtaining information about digital 
representations that contain watermarks, the method being 
employed in a network and comprising the steps of: 

sending an agent that includes a program that reads 
watermarks via the network to a system on which 
digital representations with watermarks may be resi- 
dent; and 

receiving a message from the agent's program via the 
network indicating results of the agent's program's 
reading of watermarks in the system. 

31. The method set forth in claim 30 wherein: 
the watermark includes program code; and 

the agent's program causes the watermark's program code 
to be executed in the system. 

32. The method set forth in claim 30 wherein: 

the agent's program further sends the agent to a further 
system via the network. 

33. The method set forth in claim 32 wherein: 

the agent further includes a map of nodes in the network 

to be visited by the agent; and 
the agent's program selects the further system as specified 

in the map. 

34. The method set forth in claim 30 wherein: 

the agent further includes a description of the digital 
representations whose watermarks are to be read by the 
agent. 

35. The method set forth in claim 30 wherein: 

the agent further includes a first key used to encrypt the 
message. 

36. The method set forth in claim 30 wherein: 
the watermark is encrypted; and 

the agent further includes a second key used to decrypt the 
watermark. 

37. The method set forth in claim 30 wherein 

the system includes a watermark engine to which the 
agent is sent; and the method further comprises the step 
of: 

encrypting the agent so that it can only be decrypted by 
the watermark engine. 

38. The method set forth in any one of claims 30 through 
37 wherein: 
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the message indicates a violation of security policy with 
regard to a digital representation resident on the system 
whereby the agent is used to enforce security policy. 

39. The method set forth in any one of claims 30 through 
37 wherein: 

the message indicates that a digital representation resident 
on the system at least appears to be an unauthorized 
copy. 

40. The method set forth in claim 39 wherein: 

the message indicates that the copy at least appears to be 
an unauthorized copy of a copyrighted work, whereby 
the agent is used to enforce an owner's copyright rights 
in the copy. 

41. The method set forth m claim 39 wherein: 

the message indicates that the copy at least appears to 
violate a security policy. 

42. A method whereby a system which has received 
digital representations that contain watermarks may provide 
information about the received digital representations, the 
method being employed in a network and comprising the 
steps of: 

receiving an agent in the system via the network, the agent 
including a program that reads watermarks and sends a 
message reporting results thereof via the network; and 

executing the agent's program in the system. 

43. The method set forth in claim 42 wherein: 

the agent further includes a description of the digital 
representations whose watermarks are to be read by the 
agent; and 

the agent's program selects the digital representations as 
specified in the description therefor. 

44. The method set forth in claim 42 wherein the system 
includes a watermark engine to which the agent is sent and 
the method further comprises the step of: 

encrypting the agent so that it can only be decrypted by 
the watermark engine. 

45. The method set forth in claim 42 wherein: 

the agent's program further sends the agent to a further 
system via the network. 

46. The method set forth in claim 45 wherein: 

the agent further includes a map of nodes in the network 

to be visited by the agent; and 
the agent's program selects the further system as specified 

in the map. 

47. A method of providing information about digital 
representations containing watermarks in a system which 
has access to a network and includes a watermark agent, the 
watermark agent including a program for reading a water- 
mark and sending a message reporting a result thereof via 
the network, the method comprising the steps of: 

receiving a digital representation that has a watermark and 
that was not fetched into the system by or for the 
watermark agent; and 

executing the watermark agent's program. 

48. The method set forth in any one of claims 42 through 
47 wherein: 

the watermark includes program code; and 
the agent's program causes the watermark's program code 
to be executed in the system. 

49. The method set forth in any one of claims 42 through 
47 wherein: 

the agent further includes a first key used to encrypt the 
message. 

50. The method set forth in any one of claims 42 through 
47 wherein: 
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the watermark is encrypted; and the agent further includes 
a second key used to decrypt the watermark. 

51. The method set forth in any one of claims 42 through 
47 wherein: 

the agent's program further performs an operation on the 
digital representation containing the watermark. 

52. The method set forth in claim 51 wherein: 

the operation is one or more of changing the access rights 
to the digital representation, moving the digital 
representation, or deleting the digital representation. 

53. The method set forth in any one of claims 42 through 
47 wherein: 

the message indicates a violation of security policy with 
regard to a digital representation resident on the 
system, whereby the agent is used to enforce security 
policy. 
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54. The method set forth in any one of claims 42 through 
47 wherein: 

the message indicates that a digital representation resident 
on the system at least appears to be an unauthorized 
copy. 

55. The method set forth in claim 54 wherein: 

the message indicates that the copy at least appears to be 
an unauthorized copy of a copyrighted work, whereby 
the agent is used to enforce an owner's copyright rights 
in the copy. 

56. The method set forth in claim 54 wherein: 

the message indicates that the copy at least appears to 
violate a security policy. 
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